Response code for Chinese spiders? [closed]
Posted
by
pt2ph8
on Server Fault
See other posts from Server Fault
or by pt2ph8
Published on 2012-10-30T16:44:53Z
Indexed on
2012/10/30
17:04 UTC
Read the original article
Hit count: 160
My server is being "attacked" by Chinese spiders that don't respect the rules in my robots.txt. They are being very aggressive and using a lot of resources, so I'm going to set up some rules in nginx to block them by user agent.
Question: which response code should I return, 403, 444 (empty response in nginx) or something else? I'm wondering how the spiders will react to different status codes. What's the best practice?
© Server Fault or respective owner